Fixes to resample over DST boundaries #9623

rockg · 2015-03-10T01:24:57Z

closes #5172
closes #8744
closes #8653
closes #9173
closes #9468
closes #8794

fixes all issues but one attached to #5172. There were several issues as mentioned by my comments in the master list. They are:

Offset classes were not working properly over DST boundaries. I added flags on the class to handle specially.
normalize_date does not work with DST change dates as the tz offset is not adjusted. normalize is added to Timestamp as discussed in ENH: add .normalize() to Timestamp? #8794 (I don't like how this was done, but it seemed the easiest rather than duplicating lots of code in tslib.date_normalize.
_adjust_dates_anchored was not localizing properly once times were adjusted.

rockg · 2015-03-10T01:27:36Z

The outstanding issue is #9119 which is timezone related, but not DST related.

jreback · 2015-03-10T01:41:50Z

@rockg awesome!

I listed all of the issues, pls give a check.

pls give a release note (listing all of the issues) as well.

jreback · 2015-03-10T01:42:42Z

@rockg can you run a perf test (looks like it should be fine, but pls check)

jreback · 2015-03-10T23:21:04Z

@rockg I'd like to get this in the 0.16.0 rc, which I am going to do prob tomorrow (0.16. prob 1 week later), can you update?

rockg · 2015-03-11T00:05:24Z

Yes, fixing everything up. Should have something in an hour.

rockg · 2015-03-11T01:23:03Z

Wow that vbench takes over an hour (and it only works for python 2 which was a nice surprise). There are a few date things in here which surprised me.

reshape_stack_simple                         |   3.8433 |   3.5164 |   1.0930 |
frame_ctor_dtindex_BusinessDayx1             |   2.2870 |   2.0920 |   1.0932 |
dataframe_resample_min_numpy                 |   3.2500 |   2.9697 |   1.0944 |
frame_ctor_dtindex_CustomBusinessDayx1       |   2.3030 |   2.1026 |   1.0953 |
frame_ctor_dtindex_BDayx2                    |   2.3340 |   2.1287 |   1.0965 |
frame_ctor_dtindex_YearBeginx1               |   2.2504 |   2.0523 |   1.0965 |
frame_ctor_dtindex_CBMonthEndx1              |   6.5024 |   5.8760 |   1.1066 |
groupby_ngroups_100_sum                      |   0.7041 |   0.6344 |   1.1099 |
groupby_ngroups_100_count                    |   0.5260 |   0.4703 |   1.1183 |
reshape_unstack_simple                       |   5.0033 |   4.4693 |   1.1195 |
concat_empty_frames2                         |   1.4807 |   1.3180 |   1.1234 |
timeseries_custom_bmonthend_incr             |   0.3620 |   0.3190 |   1.1348 |
panel_shift_minor                            |   0.1633 |   0.1436 |   1.1372 |
replace_replacena                            |   2.7843 |   2.4397 |   1.1413 |
timeseries_custom_bmonthbegin_decr_n         |   0.4679 |   0.4090 |   1.1442 |
replace_fillna                               |   3.4404 |   3.0044 |   1.1451 |
timeseries_iter_periodindex_preexit          |  14.1080 |  12.3093 |   1.1461 |
timeseries_custom_bmonthbegin_incr_n         |   0.4480 |   0.3854 |   1.1625 |
strings_lower                                |   5.3800 |   4.4383 |   1.2122 |
timeseries_custom_bday_cal_decr              |   0.0687 |   0.0520 |   1.3211 |
timeseries_custom_bday_decr                  |   0.0679 |   0.0513 |   1.3235 |
timeseries_custom_bday_cal_incr_neg_n        |   0.0690 |   0.0517 |   1.3354 |
timeseries_custom_bday_cal_incr              |   0.0617 |   0.0447 |   1.3808 |
timeseries_custom_bday_cal_incr_n            |   0.0620 |   0.0447 |   1.3879 |
timeseries_year_apply                        |   0.0453 |   0.0319 |   1.4179 |
timeseries_year_incr                         |   0.0490 |   0.0337 |   1.4528 |
timeseries_custom_bday_apply_dt64            |   0.0520 |   0.0356 |   1.4598 |
timeseries_custom_bday_incr                  |   0.0477 |   0.0320 |   1.4888 |
timeseries_custom_bday_apply                 |   0.0456 |   0.0306 |   1.4909 |
frame_object_equal                           |  21.1930 |  11.3687 |   1.8642 |
frame_nonunique_equal                        |  21.2193 |  11.3026 |   1.8774 |

jreback · 2015-03-11T01:32:17Z

vbench looks fine, the things that take < 1ms are really hard to time correctly.
usually I pass -r regex which will only do those tests that are most interesting.

rockg · 2015-03-11T01:33:28Z

Great news.

jreback · 2015-03-11T01:35:18Z

I think this is ok, however, need to check one thing. Can you make a pickle from 0.15.2 and add to the tests, like this: https://github.com/pydata/pandas/blob/master/pandas/tests/test_categorical.py#L2640 (need to test that the DateOffsets survive this change). Basically save say a dict of a couple of examples, pickle it to a file (in 0.15.2), then read in (in the tests) and make sure they compare equal (I think they should), but since adding this attribute, need to make sure.

rockg · 2015-03-11T02:57:35Z

The pickle test worked fine. Think we should be good on that front.

jreback · 2015-03-11T03:00:44Z

ok looks good

will merge in the morning

if u have s chance see if anything needs to be updated in the docs about tz (eg in resample section)

changes to offset classes that weren't working over such boundaries as well as adding normalize() on Timestamp.

jreback · 2015-03-11T12:10:07Z

merged via dcc68d7

thanks!

rockg · 2015-03-11T13:06:23Z

@jreback...thanks. There are a couple travis errors on two of 2.7 jobs with the pickle test (however one of them succeeded). Any thoughts?

======================================================================
ERROR: test_pickle_v0_15_2 (pandas.tseries.tests.test_offsets.TestCommon)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tseries/tests/test_offsets.py", line 424, in test_pickle_v0_15_2
    self.assertEqual(offsets, read_pickle(pickle_path))
  File "/home/travis/build/pydata/pandas/pandas/io/pickle.py", line 60, in read_pickle
    return try_read(path)
  File "/home/travis/build/pydata/pandas/pandas/io/pickle.py", line 57, in try_read
    return pc.load(fh, encoding=encoding, compat=True)
  File "/home/travis/build/pydata/pandas/pandas/compat/pickle_compat.py", line 116, in load
    return up.load()
  File "/home/travis/miniconda/envs/pandas/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/travis/build/pydata/pandas/pandas/compat/pickle_compat.py", line 20, in load_reduce
    stack[-1] = func(*args)
  File "/home/travis/miniconda/envs/pandas/lib/python2.6/copy_reg.py", line 48, in _reconstructor
    obj = object.__new__(cls)
TypeError: object.__new__(X): X is not a type object (classobj)

jreback · 2015-03-11T13:16:47Z

I rebuilt the pick on 0.15.2 and it works fine. How did you create the pickle?

I have only seen this error IIRC if you pickle I think in text mode.

rockg · 2015-03-11T13:34:39Z

I thought I used the same recipe as your example. I created a new virtualenv and saved the pickle with the file in 'wb' mode.

jreback · 2015-03-11T13:37:33Z

Python 2?

rockg · 2015-03-11T14:17:14Z

Almost positive (I can confirm tonight). I am positive that I ran the test locally with Python 2.7. Anyways, I'm not sure it's worth more thought right now. Thanks for taking care of it.

jreback added Bug Timezones Timezone data dtype Resample resample method labels Mar 10, 2015

jreback added this to the 0.16.0 milestone Mar 10, 2015

rockg force-pushed the dst-date branch from e9c5d65 to f27ca68 Compare March 11, 2015 01:22

rockg force-pushed the dst-date branch from f27ca68 to a482d0a Compare March 11, 2015 01:27

rockg force-pushed the dst-date branch from a482d0a to ff16d89 Compare March 11, 2015 02:56

Fixes to resample over DST boundaries. This requires

68e862c

changes to offset classes that weren't working over such boundaries as well as adding normalize() on Timestamp.

rockg force-pushed the dst-date branch from ff16d89 to 68e862c Compare March 11, 2015 11:45

jreback closed this Mar 11, 2015

jreback mentioned this pull request Apr 12, 2015

Resample BM adds extra index point #9756

Closed

JackKelly mentioned this pull request May 12, 2015

resample gives AmbiguousTimeError when index ends on a DST boundary #10117

Closed

rockg mentioned this pull request Dec 28, 2017

Unclear logic for which offsets are _adjust_dst #18977

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes to resample over DST boundaries #9623

Fixes to resample over DST boundaries #9623

rockg commented Mar 10, 2015

rockg commented Mar 10, 2015

jreback commented Mar 10, 2015

jreback commented Mar 10, 2015

jreback commented Mar 10, 2015

rockg commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015 via email

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

Fixes to resample over DST boundaries #9623

Fixes to resample over DST boundaries #9623

Conversation

rockg commented Mar 10, 2015

rockg commented Mar 10, 2015

jreback commented Mar 10, 2015

jreback commented Mar 10, 2015

jreback commented Mar 10, 2015

rockg commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015 via email

jreback commented Mar 11, 2015

rockg commented Mar 11, 2015